System for endoscopic imaging and method for processing images

ABSTRACT

The invention relates to a system for endoscopic imaging comprising a light source configured to generate light and an, in particular rigid, insertion portion configured to be inserted into an object and comprising a distal end, a proximal end and at least one light guiding path. The system further comprises a first imaging device mounted at the proximal end of the insertion portion and optically coupled to the light guiding path, the first imaging device comprising a plurality of first detecting elements exhibiting a first sensitivity to light. A flexible guiding portion comprising a distal end and a proximal end is provided to guide a second part of the light emanating from the medium from the distal end of the guiding portion to the proximal end of the guiding portion. A second imaging device provided at the proximal end of the flexible guiding portion comprises a plurality of second detecting elements exhibiting a second sensitivity to light, the second sensitivity of the second detecting elements being higher than the first sensitivity of the first detecting elements. A control unit is configured to derive at least one third image of the medium based on image data of the at least one first image of the medium and image data of the at least one second image of the medium. The invention further relates to a corresponding method for processing images.

The present invention relates to a system for endoscopic imaging and acorresponding method for processing images.

Optical imaging is widely employed in medicine during endoscopicprocedures, e.g. in surgery, ophthalmology and other applications.Healthy and diseased tissues exhibit differences in several properties,e.g. structural, compositional, metabolic, molecular and cellularproperties, which can be optically detected. Detection is based eitheron intrinsic tissue chromophores such as haemoglobin or melanin orextrinsically administered agents (e.g. fluorochromes) which can in-vivostain physiological, cellular or molecular features (e.g. perfusion,permeability, inflammation, receptor distribution etc). Local orsystemic administration of agents with specificity to cellular andsubcellular tissue and disease biomarkers can alter the opticalproperties of healthy and diseased tissue in different ways resulting invisualization of lesions with high contrast with the background healthytissues. Recent studies indicate that the use of externally administeredfluorescent agents is a highly promising approach since fluorescencesignals can provide high contrast. For example, engineered agents can bevery sensitive and specific in cancer detection by targeting specificmolecular features of carcinogenesis and tumor lesions. This informationcan be combined with contrast from intrinsic tissue chromophores likehaemoglobin or melanin to yield accurate predictive, diagnostic and/orinterventional guidance.

The majority of optical imaging systems employed clinically today arebased on a photographic or video approach employing reflectance imaging,also termed epi-illumination. The term fluorescence imaging is alsobroadly employed to describe photography or video approaches using afilter to reject excitation light and register only fluorescence photonson the camera. Often two or multiple camera systems are employed, somecameras detecting color images, some cameras fluorescence images.

Other designs have been proposed that utilize imaging at multiplewavelengths, also referred to as spectral imaging, multispectral imagingor hyperspectral imaging, which can be employed to resolve differentoptical absorbers or differentiate auto-fluorescence from a fluorochromeof interest.

The invention is based on the problem to provide an improved system forendoscopic imaging and method for processing images, in particularallowing for both high sensitivity imaging and easy handling.

The problem is solved by the system and method according to theindependent claims.

A system for endoscopic imaging according to an aspect of the inventioncomprises a light source configured to generate light and an insertionportion configured to be inserted into an object and comprising a distalend, a proximal end and at least one light guiding path configured toguide the generated light to the distal end of the insertion portion andto guide light emanating from a medium within the object to the proximalend of the insertion portion. The system further comprises a firstimaging device mounted at the proximal end of the insertion portion andoptically coupled to the light guiding path, the first imaging devicecomprising a plurality of first detecting elements, the first detectingelements exhibiting a first sensitivity to light and being configured todetect a first part of the light emanating from the medium to obtain afirst image of the medium exhibiting a first spatial resolution.Further, a flexible guiding portion comprising a distal end and aproximal end is provided, the distal end of the guiding portion beingoptically coupled to the light guiding path. The guiding portion isconfigured to guide a second part of the light emanating from the mediumfrom the distal end of the guiding portion to the proximal end of theguiding portion. Additionally, a second imaging device is provided atthe proximal end of the flexible guiding portion and comprises aplurality of second detecting elements, the second detecting elementsexhibiting a second sensitivity to light and being configured to detectthe second part of the light emanating from the medium to obtain asecond image of the medium exhibiting a second spatial resolution, thesecond sensitivity of the second detecting elements being higher thanthe first sensitivity of the first detecting elements.

In a method according to another aspect of the invention, imagesgenerated by a system for endoscopic imaging are processed, wherein thesystem comprises a light source configured to generate light, an, inparticular rigid, insertion portion configured to be inserted into anobject and comprising a distal end, a proximal end and at least onelight guiding path configured to guide the generated light to the distalend of the insertion portion and to guide light emanating from a mediumwithin the object to the proximal end of the insertion portion, a firstimaging device mounted at the proximal end of the insertion portion andoptically coupled to the light guiding path, the first imaging devicecomprising a plurality of first detecting elements, the first detectingelements exhibiting a first sensitivity to light and being configured todetect a first part of the light emanating from the medium to obtain atleast one first image of the medium exhibiting a first spatialresolution, a flexible guiding portion comprising a distal end and aproximal end, the distal end of the guiding portion being opticallycoupled to the light guiding path, and being configured to guide asecond part of the light emanating from the medium from the distal endof the guiding portion to the proximal end of the guiding portion, and asecond imaging device provided at the proximal end of the flexibleguiding portion and comprising a plurality of second detecting elements,the second detecting elements exhibiting a second sensitivity to lightand being configured to detect the second part of the light emanatingfrom the medium to obtain at least one second image of the mediumexhibiting a second spatial resolution, the second sensitivity of thesecond detecting elements being higher than the first sensitivity of thefirst detecting elements, and wherein the method comprises deriving atleast one third image of the medium based on image data of the at leastone first image of the medium and image data of the at least one secondimage of the medium by

correcting image data of the at least one second image of the medium formotion of the insertion portion and the medium relative to each otherbased on image data of the at least one first image of the medium,and/or

combining, in particular fusing, image data of the at least one firstimage of the medium and image data of the at least one second image ofthe medium to obtain the at least one third image, wherein the range, inparticular a ratio or difference, between a highest image data value anda lowest image data value contained in the image data of the at leastone third image is larger than the range between a highest image datavalue and a lowest image data value contained in each of the image dataof the at least one first image and the image data of the at least onesecond image.

Preferably, the system for endoscopic imaging provides a first and asecond imaging device which are arranged spatially separated from eachother. In particular, the first imaging device, e.g. a first cameraexhibiting a first sensitivity to light, is arranged at a proximal endof an insertion portion, e.g. an endoscope, which is configured to be atleast partially inserted into an object, and optically coupled to alight guiding path of the insertion portion configured to convey lightfrom a distal end of the insertion portion to the proximal end of theinsertion portion or vice versa. The second imaging device, e.g. asecond camera exhibiting a second sensitivity to light which is higherthan the first sensitivity of the first camera, is arranged at aproximal end of a flexible guiding portion, the distal end of whichbeing optically coupled to the light guiding path of the insertionportion as well. Thereby, light generated by a light source, which ispreferably arranged in the region of the second imaging device andoptically coupled to the proximal end of the flexible guiding portion aswell or, alternatively, arranged in the region of the first imagingdevice and optically coupled to the light guiding path, is conveyed bythe light guiding path to a medium inside the object, and lightemanating, e.g. due to reflection, scattering and/or luminescence, fromthe medium is conveyed to the first imaging device and to the secondimaging device without the necessity of the second imaging device and,if applicable, the light source, being arranged at the proximal end ofthe insertion portion.

By means of the flexible guiding portion the first imaging device andthe second imaging device are spatially separated from each other, sothat even a heavy and bulky high-sensitivity second imaging device canbe included into the system without limiting the handiness and/orhandling of the insertion portion and the available space at theproximal end of the insertion portion.

Further, the arrangement described above allows for simultaneousacquisition of images by the first and the second imaging device suchthat real-time imaging can be performed.

Moreover, providing two different imaging devices for endoscopicimaging, wherein the second imaging device is not limited by size orweight, allows for flexible imaging and provides a user, e.g. aphysician, with increased information. For example, a first, lightweightcamera which is configured to generate high-resolution color images canbe directly optically coupled to the light guiding path of alaparoscope, e.g. by mounting the first camera at the proximal end ofthe light guiding path, and a second, heavyweight camera configured tocapture fluorescence images at high sensitivity can be indirectlyoptically coupled to the light guiding path via the flexible guidingportion. Because the second camera is not directly mounted at thelaparoscope, the distal end of the laparoscope can be inserted into theobject, e.g. a human body, easily and positioned therein with highprecision, regardless of the weight and/or size of the second camera.Light generated by the light source is conveyed through the endoscopeand illuminates tissue in a region of the distal end of the laparoscope,such that light emanating from the tissue in response thereof isdetected by the first and the second camera, preferably simultaneously.

In summary, the invention provides an improved system for endoscopicimaging and method for processing images, in particular allowing forhigh sensitivity imaging and easy handling.

Within the meaning of the invention, the term “endoscopic imaging”relates to any kind of imaging in which an inward or interior part of anorganism and/or any object or cavity is imaged. For example, endoscopicimaging includes, but is not limited to, laparoscopic, thoracoscopic,esophagoscopic, gastroscopic, coloscopic and arthroscopic imaging.

According to a preferred embodiment of the invention, the first spatialresolution of the first image is higher than the second spatialresolution of the second image. In particular, the first imaging devicecomprises an imaging sensor, which is particularly lightweight andprovides a particularly high spatial resolution at small dimensions. Bythis means, optical images of the medium of the object with both highsensitivity (via second imaging device) and high spatial resolution (viafirst imaging device) can be obtained simultaneously.

According to another preferred embodiment of the invention, the systemcomprises a coupling element provided at the proximal end of theinsertion portion, the coupling element being configured to opticallycouple both the first imaging device and the distal end of the flexibleguiding portion to the light guiding path of the insertion portion. Bythis means, the first part of the light can be easily and reliablyguided from the light guiding path to the first imaging device, while atthe same time the second part of the light can be easily and reliablyguided from the light guiding path, via the flexible guiding portion, tothe second imaging device. In particular, the coupling element allowsfor simultaneous transmission of the light emanating from the lightguiding path at the proximal end of the insertion portion towards thefirst imaging device and the second imaging device such that real-timeimaging with particularly high sensitivity and spatial resolutionbecomes possible.

According to yet another embodiment of the invention, the couplingelement comprises a semi-transparent or dichroic mirror configured toreflect or transmit the first part of the light emanating from themedium to the first detecting elements, and to transmit or reflect,respectively, the second part of the light emanating from the medium tothe second detecting elements. This is particularly advantageous if thefirst part of the light comprises a different polarization than thesecond part of the light or if the first part of the light comprises adifferent wavelength or spectrum of wavelengths than the second part ofthe light. This allows for particularly reliable separation of the firstpart of the light from the second part of the light to be detected bythe first and second imaging device, respectively.

According to yet another embodiment of the invention the insertionportion is rigid. Because of the lightweight low-sensitivity firstcamera mounted at the proximal end of the rigid insertion portion, theinsertion portion is also lightweight and slim and can be easily handledand inserted into a soft object, such as a human or animal body, and/orprecisely positioned in the soft object such that areas of interest,e.g. where physician suspects diseased tissue, can be imaged reliably.

Alternatively, the insertion portion or at least a part thereof isflexible. Likewise, the lightweight low-sensitivity first camera mountedat the proximal end of the flexible insertion portion allows for an easyhandling of the lightweight and slim insertion portion, e.g. wheninserting the insertion portion into wound or crooked cavities of anobject as with gastroscopy or coloscopy, and reliable imaging of theinner structures of such cavities.

In yet another embodiment of the invention, the first part of the lightemanating from the medium corresponds to light reflected by the mediumin response to an irradiation of the medium with the generated light.Preferably, the first imaging device comprises an optical filterconfigured to transmit light being reflected by the medium and tosuppress all other light emanating from the medium, e.g. due to itsspecific polarization or wavelength. By this means, the sensitivity ofthe first imaging device for light reflected by the medium can beincreased such that high quality reflection images of the medium in theobject can be reliably acquired.

In particular, the second part of the light reflected by the mediumcorresponds to a diagnostic image-forming signal, including but notlimited to a, in particular high resolution, color image.

In yet another embodiment of the invention, the second part of the lightemanating from the medium corresponds to luminescence, Raman-scatteredor scintillation light emitted by the medium in response to anirradiation of the medium with the generated light or radiation.Preferably, the second imaging devices comprises an optical filterconfigured to transmit luminescence, Raman-scattered or scintillationlight emanating from the medium and to suppress other light emanatingfrom the medium, e.g. due to its specific polarization or wavelength. Bythis means, the sensitivity of the second imaging device forluminescence, Raman-scattered or scintillation light emanating from themedium can be increased such that high-quality luminescence orscintillation images, respectively, of the medium in the object can bereliably acquired.

Within the meaning of present invention, the term “luminescence”includes, but is not limited to, fluorescence, phosphorescence,bioluminescence, chemoluminescence, Raman emission, radio-luminescence(e.g. Cherenkov radiation).

In particular, the second part of the light emanating from the mediumcorresponds to a diagnostic image-forming signal including, but notlimited to, luminescence or light emanating from a scintillator incorrespondence to incident nuclear radiation.

In yet another embodiment of the invention, the light guiding path ofthe insertion portion comprises a tubular optical path and at least onerelay lens configured to relay an image of the medium through thetubular optical path to the first detecting elements of the firstimaging device. Preferably, the first relay lens is arranged and/ormounted at the proximal end of the insertion portion and/or the distalend of the insertion portion and images the medium onto the firstdetecting elements via the coupling element. The tubular optical pathmay comprise a coherent fiber bundle. Alternatively, the tubular opticalpath comprises a cavity, preferably filled with a coupling medium, e.g.gas and/or liquid, to reliably convey the light emanating from themedium in response to illumination of the medium with light. By thismeans, the medium near the distal end of the insertion portion can bereliably and precisely imaged by the first imaging device.

In yet another embodiment of the invention, the flexible guiding portioncomprises an optical fiber bundle, preferably a coherent optical fiberbundle, configured to relay an image of the medium through the opticalfiber bundle to the second detecting elements of the second imagingdevice. Preferably, the second imaging device is configured tocompensate for a possible loss of intensity of the second part of thelight in the fiber bundle with an increased sensitivity of the secondimaging device. By this means, the insertion portion can be easily andflexibly handled without adding the weight of the second imaging deviceto the insertion portion, while optical images can be obtained by thesecond imaging device with high sensitivity.

According to another aspect of the invention, the system furthercomprises a control unit configured to derive a third image of themedium based on image data of the first image of the medium and imagedata of the second image of the medium. The third image may containinformation of both the first and the second image and/or informationderived from a combination of the first and the second image. In thisway, third images with enhanced diagnostic conclusiveness can beobtained.

According to yet another aspect of the invention, the control unit isfurther configured to derive the third image of the medium by correctingimage data of the, preferably high-sensitivity, second image of themedium for a possible motion of the insertion portion and the mediumrelative to each other based on image data of the, preferablyhigh-resolution, first image of the medium. By this means, particularlyconclusive diagnostic images, which do not suffer from motion blur orartifacts, of the medium are obtained.

Preferably, the control unit is configured to determine, in particularestimate, a two-dimensional motion field, which determines the velocityof the medium relative to the insertion portion or velocities ofcomponents and/or different regions of the medium relative to eachother, by means of the, in particular high-resolution, first image. Thecontrol unit is preferably further configured to adapt the second image,in particular obtained with second detecting elements with highsensitivity, according to or based on the two-dimensional motion field.Additionally, the adapted second image may be further combined withsuccessively obtained further second images, by which means secondimages may be accumulated such that the exposure time of the secondimages may be virtually prolonged.

Preferably, the control unit is further configured to determine, basedon image data of the at least one first image of the medium, at leastone two-dimensional motion field characterizing a velocity of the mediumrelative to the insertion portion or velocities of different regions ofthe medium relative to each other.

Preferably, the control unit is further configured to revert amotion-induced relative displacement of image data of the at least onesecond image based on the at least one two-dimensional motion field toobtain the at least one third image of the medium.

Preferably, the control unit is further configured to revert amotion-induced relative displacement of image data in each of two ormore successively obtained second images based on the at least onetwo-dimensional motion field to obtain two or more adapted secondimages, and to combine the two or more adapted second images to obtainthe at least one third image of the medium.

Preferably, the control unit is further configured to determine, basedon image data of at least two successively obtained first images of themedium, a two-dimensional motion field, to revert a motion-inducedrelative displacement of image data in each of at least two successivelyobtained second images based on the two-dimensional motion field toobtain two or more adapted second images, and to combine the two or moreadapted successively obtained second images to obtain the at least onethird image of the medium.

Preferably, the motion field refers or corresponds to a description ofmotion-induced vector components, in particular regarding a distance anddirection, of every picture element or image element (pixel) between twosubsequently acquired first images.

Preferably, the control unit is further configured to weight image dataof the second image with a factor α the magnitude of which depending onhow much recent image data of the second image is, wherein image data ofa more recently obtained second image is weighted with a higher factorthan image data of a less recently obtained second image.

Preferably, the at least one first image of the medium being a colorimage and/or reflectance image of the medium, and the at least onesecond image of the medium being a fluorescent image of a fluorescenceagent contained in the medium.

According to an alternative or additional aspect of the invention, thecontrol unit is further configured to derive the third image of themedium by combining, in particular fusing, image data of the first imageof the medium and image data of the second image of the medium. Forexample, by combining image data of a high-resolution color image oftissue (corresponding to the first image) with image data of afluorescent image of a fluorescence agent (corresponding to the secondimage), a physician can easily identify disease tissue to which thefluorescence agent specifically binds. By this means, the dynamic rangeof the third image can be increased compared to the dynamic range of thefirst or second image.

Preferably, the term “dynamic range” refers to a range, a ratio or adifference between a highest image data value and a lowest image datavalue contained in the image data of the at least one first, second orthird image, respectively. Accordingly, the range, ratio or differencebetween the highest image data value and the lowest image data valuecontained in the image data of the at least one third image is largerthan the range, ratio or difference between the highest image data valueand lowest image data value contained in each of the image data of theat least one first image and the image data of the at least one secondimage.

For example, image data value(s) contained in the image data of animage, in particular of the first, second and third image, may be pixelvalue(s), grey-level or grey scale value(s) and/or color or tonalvalue(s) at the pixel(s) of the image.

Preferably, the control unit is configured to spatially registerregions, in particular pixels, of the first and second images with eachother and to combine the regions such that high-sensitivity andhigh-resolution information complement each other in each of thecorresponding regions in the third image.

Preferably, the control unit is configured to control the first imagingdevice and the second imaging device to detect the first part of thelight emanating from the medium and the second part of the lightemanating from the medium simultaneously.

Preferably, the first part of the light and the second part of the lightemanate from the same region of interest of the medium or fromoverlapping regions of interest of the medium.

It is noted that the preferred embodiments of the system described aboveaccordingly apply to the method for processing images. Thus, it ispreferred that the method for processing images further comprises and/orperforms one or more steps corresponding to one or more steps and/orfunctions executed by the control unit, i.e. steps and/or functions thecontrol unit is configured to execute, as described above.

Further advantages, features and examples of the present invention willbe apparent from the following description of following figures:

FIG. 1 shows an example of a system for endoscopic imaging;

FIG. 2 shows an exemplary schematic of a system for endoscopic imaging;

FIG. 3 shows an exemplary schematic of a first imaging mode; and

FIG. 4 shows an exemplary schematic of a second imaging mode.

FIG. 1 illustrates an exemplary system 1 for endoscopic imagingcomprising a light source 2, an insertion portion 3, a first imagingdevice 4, a flexible light guiding portion 5 and a second imaging device6.

The insertion portion 3 is configured to be inserted, at leastpartially, in particular with a distal end 3 b, into an object 9.Preferably, the object 9, also referred to as sample, is a biologicalobject, in particular a human or animal body or a part thereof. Inparticular, the sample comprises a medium 11, e.g. biological tissue ora part thereof. Accordingly, the system is particularly suited formedical imaging.

The light guiding portion 5, e.g. a coherent fiber bundle, is configuredto convey light generated by the light source 2 to the insertion portion3. Preferably, the light source 2 is optically coupled via a couplingdevice 7, e.g. a lens, to a proximal end 5 a of the light guidingportion 5.

At its distal end 5 b, the light guiding portion 5 is optically coupledto a light guiding path 10, e.g. an illumination port of a laparoscope,of the insertion portion 3 via a coupling element 8, e.g. asemi-transparent or dichroic encoded beam-splitter. The light guidingpath 10 is configured to convey light from a proximal end 3 a of theinsertion portion 3 to the distal end 3 b of the insertion portion 3 orvice versa.

The light generated by the light source 2 is transmitted by the couplingelement 8 and conveyed through the light guiding path 10 such that it isemitted at the distal end 3 b of the insertion portion 3, therebyilluminating the medium 11 of the object 9.

In response to the illumination, light emanates from the medium 11.Emanating light may be, e.g., reflected and/or scattered light and/orluminescent light which is excited in the medium 11 in response to theillumination. At least a part of the emanating light re-enters orenters, respectively, at the distal end 3 b of the insertion portion 3and is guided through the optical path 10 to the coupling element 8.

The coupling element 8 is preferably configured to split the lightemanating from the medium 11 into a first part and a second part,wherein the first part of the light emanating from the medium 11 isrelayed to the first imaging device 4 by means of a relay lens 12. Thefirst imaging device 4 comprises a plurality of detecting elements 4 awhich are configured to detect the first part light to obtain a firstimage of the medium 11, preferably at a high spatial resolution.

In some embodiments, the coupling element 8 is configured to split thelight emanating from the medium 11 dependent on properties of the lightemanating from the medium 11, e.g. on photon energy or polarization, ora combination thereof.

The second part of the light emanating from the medium 11 is relayed tothe second imaging device 6 by means of the flexible guiding portion 5.At the proximal end 5 a of the flexible guiding portion 5, the secondpart of the light is detected by a plurality of detecting elements 6 aof the second imaging device 6 such that a second image can be obtained,in particular with high sensitivity.

Additionally, further imaging devices can be arranged at the proximalend 3 a of the insertion portion 3, i.e. optically coupled to the lightguiding path 10 by means of the coupling element 8, and/or at theproximal end 5 a of the flexible guiding portion 5 for obtainingmultiple images of the medium 11.

Preferably, the first and second imaging devices 4, 6 are opticalcameras, i.e. photon detection sensors, e.g. a charged coupled device(CCD), a complementary metal-oxide-semiconductor (CMOS) sensor, anindium gallium arsenide (InGaAs) sensor. Preferably, the photondetection sensors, e.g. the CCD, CMOS and/or InGaAs sensors, are cooled.

In some embodiments, each of the first and second imaging devices 4, 6may be constituted by more than one of the above-mentioned photondetection sensor types. In particular, a first plurality of detectingelements 4 a, 6 a of the first and/or second imaging device 4, 6 maycorrespond to detecting elements of a first type of the above-mentionedsensors, and a second plurality of detecting elements 4 a, 6 a of thefirst and/or second imaging device 4, 6 may correspond to detectingelements of a second type of the above-mentioned sensors. This allowsfor a flexible adaptation of the imaging devices 4, 6 to therequirements of practical applications where different signalscorresponding to different components of the light emanating from themedium 11 exhibit different signal strengths and different dynamicranges.

Preferably, the sensitivity of one or more of the independentlysensitive CCD or CMOS or InGaAs sensors or corresponding detectorelements, respectively, is automatically adapted, in particular throughvariable attenuation or amplification of the sample signals collected orthe corresponding electrical signals.

In some embodiments, each of the at least two imaging devices 4, 6 isprovided with a field filter being adjusted to a spectral sensitivityrange of the respective imaging device 4, 6. Preferably changeable fieldfilters are provided, by which the flexibility of the system is furtherimproved.

Preferably, the light source 2 comprises at least a white lightillumination arrangement configured to generate a broad illuminationspectrum, and at least one target excitation illumination arrangementconfigured to generate at least one target wavelength. The targetwavelength may be chosen from a large spectrum of wavelengths includingthe UV, visible, NIR and IR spectral regions, e.g. 0.2 μm-10 μm.Preferably, the use of excitation at NIR wavelengths, e.g. atwavelengths between 650 nm and 1100 nm, or the use of excitation at IRwavelengths between 1250 nm and 1350 nm allows seamless separation ofwhite light images and near-infrared or infrared wavelengths. Due toreducing scatter in tissue with increasing wavelength, the use offar-NIR wavelengths, e.g. between 900 nm and 1350 nm, may lead to imagesof higher resolution.

Alternatively or additionally, temporal and spatial properties of lightmay be exploited as well. In particular, the generation of lightpatterns by the light source 2, e.g. spatially modulated lightestablishing areas of higher intensity and areas of lower intensity inthe medium 11, may be utilized for achieving separation of opticalproperties, improving resolution or suppressing back-ground signals.Moreover, the generation of intensity-modulated light, e.g. lightpulses, by the light source 2 can also be utilized to suppressbackground signals or simultaneously interleave multiple wavelengths.For example, modulation in the Hz-kHz range or so-called pulseinterleaving, using pulses of different wavelengths, can allowsimultaneous imaging at multiple wavelengths.

The overlapping information of different properties of light and/ortissue are preferably encoded or decoded, respectively, intime-interlaced fashion on either detection or illumination gating.

The system 1 for endoscopic imaging as illustrated in FIG. 1 may beemployed to collect first and second images corresponding to differenttissue properties, including polarization, auto-fluorescence, orfluorescence emanating from markers administered to the medium 11 forcontrast enhancement. These images, also referred to as marker images,indicate intrinsic or extrinsic markers obtained in addition to thetraditional color (reflectance) images obtained by the first imagingdevice 4.

The first part of the light emanating from the medium 11, also referredto as reflected light, is relayed onto the first imaging device 4, whilethe second part of the light emanating from the medium 11, also referredto as marker light, is relayed through the flexible guiding portion 5onto the second imaging device 6, which is preferably particularlysensitive to the at least one wavelength of the marker light.

Preferably, both the reflection light and the marker light are collectedsimultaneously, thus allowing a real time processing of the multipledifferent images of the sample.

FIG. 2 shows an exemplary schematic of a system 1 for endoscopicimaging, which is split into light-weight image acquisition hardware 40comprising a first imaging device 4, and heavy-weight imagingacquisition hardware 60 comprising a second imaging device 6. Althoughnot shown, either one or both of the image acquisition hardware 40, 60may comprise additional imaging devices.

Preferably, the heavy-weight image acquisition hardware 60 is mounted ona mobile wheel-based rack (not shown). The image acquisition hardware40, 60 entities are connected via flexible guiding portion 5, which maycomprise a coherent optical fiber bundle and/or electrical powerconnection and/or data connection in order to optically and/orelectrically couple image acquisition hardware 40 with image acquisitionhardware 60. In term of optical coupling, above elucidations regardingthe flexible guiding portion 5 shown in FIG. 1 apply accordingly.

The system 1 further comprises a control unit 100, e.g. an integratedimage acquisition and processing device, which is preferably configuredto execute a computer program to generate digital images from image dataobtained from the first and second imaging device 4, 6 and process sameby image processing algorithms, in particular a first imaging mode and asecond imaging mode described in detail further below.

The control unit 100 preferably comprises an acquisition module 105configured to acquire image data from the first and second imagingdevice 4, 6, a first processing module 101 configured to process theacquired image data in the first and/or second imaging mode, a secondprocessing module 102 configured to combine, in particular merge, theprocessed image data, and a third imaging module 103 configured toquantify the merged image data, before single or combined, in particularmerged, images are displayed on a display unit 104.

It is noted that the image data processing steps by means of theprocessing modules 101, 102, 103 are not mandatory, i.e. it is possibleto convey acquired image data from the acquisition module 105 to thesecond processing module 102 to merge, e.g., image data of a first imageand image data of a second image without processing the image data inthe first and/or second imaging mode. Likewise, it is possible to conveyimage data to the display unit 104 without merging the image data in thesecond processing module 102.

Preferably, the control unit 100 further comprises a data storage module106 configured to store the image data acquired by means of theacquisition module 105 in a database 107 for re-evaluation,documentation purposes or training purposes.

In particular, the control unit 100 is adapted for processing firstimages, e.g. multi-spectral reflection images, and second images, e.g.marker light images, in parallel and rendering at least one combinedimage based on at least one first image and at least one second image.The at least one combined image can be processed and rendered inreal-time, i.e. with a delay after the image collection such that thedelay is negligible in terms of human visual perception, preferably witha delay of less than 500 ms, more preferably less than 100 ms, inparticular less than 50 ms.

Providing at least one combined image in real-time may also includeproviding an image sequence, i.e. a video sequence, of combined images.As an example, the control unit 100 may be configured for generating avideo sequence of at least one first image, at least one second image,at least one combined image or a combination thereof.

Additionally or alternatively, spectral collection of image data can beachieved by time-sharing the activation of multiple illuminationarrangements of a light source in time-synchronized subsequent full orpartial readouts of image data from the first and/or second imagingdevice 4, 6. Alternatively, spectral decomposition systems, such asprisms, monochromators etc. can be employed.

FIG. 3 shows an exemplary schematic of a first imaging mode, alsoreferred to as motion-compensation mode.

In the first imaging mode, image data from first images 20, alsoreferred to as reflection images, obtained by a first imaging device, inparticular with high resolution, are utilized to determine, inparticular estimate, a two-dimensional motion field characteristic of amovement of a medium relative to an insertion portion or for a movementof portions of the medium relative to further portions of the medium.

Preferably, the motion field is a dense motion field which refers to thedescription of motion-induced vector components (e.g., distance anddirection) of every picture element (pixel) between two subsequentlyacquired images. More specifically, for every picture element of theimage, a 2D vector describes, which point corresponds to itsmotion-induced related position in the previously acquired image. Morespecifically, said 2D vectors refer to a tuple of signed rationalnumbers for the differential offset in the image coordinate referencesystem.

Also, image data from second images 30, also referred to as tracerimages, obtained by a second imaging device with high sensitivity, areregistered to the image data from the first images 20.

Subsequently, the image data from the second images 30 is weighted witha, preferably constant, factor α corresponding to the magnitude to whichextent recent image data is weighted more important than less recentlyobtained image data. By applying said factor in an iterative manner, therelative contribution of the single second images decays exponentiallywith the time that has elapsed since the most recent acquisition of thesecond image.

The motion-induced changes in the weighted image data are revertedaccording to the two-dimensional motion field of a single second image30, resulting in motion-corrected image data of the single second image30, also referred to as functional image data F_(corr). Themotion-corrected image data of the single second image 30 may then beoutputted, e.g., on a display device.

Preferably, the motion-corrected image data of at least one second image30 form a third image 31. Therefore, within the meaning of presentdisclosure, the expressions “motion-corrected image data of second image30” and “motion-corrected second image 30” are used synonymously to“image data of third image 31” or “third image 31”, respectively.

Preferably, the motion-corrected image data of the single second image30 may be combined with more recent image data of a further second image30, thereby virtually prolonging the exposure time of the second imagingdevice and increasing the signal-to-noise ratio of the further secondimage 30. Preferably, this process is repeated in an iterative manner,wherein two-dimensional motion fields are calculated from the image dataof subsequently acquired first images 20. Said motion-field is utilizedto revert the motion-induced effects of the corresponding image data ofsubsequently acquired second images 30, resulting in a particularly highsignal-to-noise ratio of a motion-corrected second image 30 or thirdimage 31, respectively. In particular, the resulting corrected secondimage 31 now incorporates information of every of the previouslyacquired single second images 30 in adapted weights (e.g., exponentiallysmoothed) and therefore depicts an advantageous combination of thesensitivity-challenged and resolution-challenged information.

FIG. 4 shows an exemplary schematic of a second imaging mode, alsoreferred to as dynamic enhancement mode, wherein image data of anacquired first image 20 as shown in FIG. 4(c) and image data of anacquired second image 30 as shown in FIG. 4(a) are spatially registeredsuch that each portion of the first image 20 corresponds to a portion ofthe second image 30.

In present example, the first and the second images 20, 30 are acquiredby the first imaging device 4 and second imaging device 6 (see FIGS. 1and 2), respectively, from the same area of interest and preferably atthe same time. As apparent from FIG. 4(c), the first image 20 isacquired at high resolution such that spatial features and details, e.g.of indicated X-shaped structure with higher contrast, in the area ofinterest are resolved well. As apparent from FIG. 4(a), the second image30 is acquired at high sensitivity such that features in the area ofinterest with lower contrast, as exemplarily indicated by a dottedcircle, can be resolved well, whereas not all details of the finerX-shaped structure are resolved.

FIG. 4(d) and FIG. 4(b) show histograms (number of pixels vs. tonalvalues of the pixels) of the first image 20 and the second image 30,respectively. Due to the high sensitivity of the second imaging device,the left part of the corresponding histogram in FIG. 4(b) containscontributions to the histogram which the histogram in FIG. 4(d) of thefirst image 20 lacks. Likewise, due to the high resolution of the firstimaging device, the right part of the corresponding histogram in FIG.4(d) contains contributions to the histogram which the histogram in FIG.4(b) of the second image 30 lacks.

The image data of the first and second image may be combined to a thirdimage 32 in a manner such that information contained in the first image20 obtained at a high resolution and information contained in the secondimage 30 obtained at a high sensitivity contribute to the third image32, as shown in FIG. 4(e), according to their sensitivity-transferfunction. By this means, the first imaging device and the second imagingdevice supplement each other regarding imaging performance such asresolution and sensitivity. Accordingly, information of the first image20 and information of the second image 30 supplement each other in thethird image 32. Thus, the third image 32 is superior to the first andthe second image taken alone.

FIG. 4(f) shows a histogram of the third image 32. As the third imagecomprises information corresponding to both high resolution and highsensitivity, the histogram contains contributions both in the left partand in the right part of the histogram.

The ratio of detectable lowest signal values and highest signal values,which is also referred to as “dynamic range”, and the quantificationresolution of the analog-to-digital converter for the pixel values(e.g., 16 Bits vs. 12 Bits) is preferably superior in the second imagesensor, in comparison to the first image sensor.

Preferably, the second imaging sensor has a higher probability that anincident photon results in an electron, which can be digitized. In otherwords, the second imaging sensor has a higher sensitivity or quantumefficiency for incident photons than the first sensor.

Preferably, the first imaging sensor and the related optical path enablea superior spatial resolution, e.g., a higher number of resolvablecontrast-inducing line-pairs per length in the imaged field of view,compared to the second imaging sensor.

The combination of first and second image to a third image, in which theadvantages of the first and second imaging device are combined and/orweaknesses of the first or second imaging device are compensated,preferably affects a DC-offset, a gain, a noise-level, differences in adetection area, spectral-band-differences, and different opticalattenuation in the optical path towards the respective sensor, thesensitivity, the dynamic range, and the spatial resolution. Preferablythe combination of the first and second image 20, 30 is applied in away, such thatF _(corr) =T ₁(F ₁,par1)+T ₂(F ₂,par2),where F_(corr) is the final corrected fluorescence image, i.e. the thirdimage 32, T₁ and T₂ are transfer functions to adjust the intensityinformation of first image F₁ and second image F₂ according to a set ofparameters par₁/par₂ of the transfer function in a way that the moresuiting source of information is dominant at the given intensities.

An advantageous transfer function for deciding the weight-factors formerging F₁ and F₂ is the logistic function

${{T(x)} = \frac{1}{1 + e^{- {k{({x - x_{0}})}}}}},$wherein the set of parameters (e.g., par₁, par₂), the parameter x₀ isthe cut-point of the logistic function, and the parameter k representsits steepness.

The specific kind of the transfer functions and the correspondingparameters for the transfer functions par1 and par2 have to becalibrated for the specific combination for optical sensors, filter andbeam splitters and the like. In a preferred application in medicalimaging, the system for endoscopic imaging may be used by executing thefollowing three steps: administering one or more contrast agents orprobes, also referred to as marker substances, e.g. molecular probes;optical imaging, in particular multispectral optical imaging; andprocessing of captured images for real-time display of correctedinformation.

The administration step is an optional step. It can be omitted, inparticular, if the sample already includes at least one marker substancefor natural reasons or due to a previous treatment. The at least onefirst image obtained with the system is also termed “inspection image”.The term “inspection image” refers to the fact that the image can beused for finding a particular tissue feature, for diagnosis, for guidingtreatment e. g. by a physician and/or by a subsequent image evaluation,or for identification of a suspicious lesion such that efficientguidance and intervention with high specificity can be provided, e.g. anintervention with therapeutic intend.

The at least one second image obtained with the system is also termed“diagnostic image”. The diagnostic image may include a map of the objecthighlighting various object conditions. Similarly the diagnostic imagecan be used to guide minimally invasive surgical intervention orendoscopically administered biopsies. However, the diagnostic image assuch preferably does not deliver the diagnosis.

The term “marker substance” refers to any molecule which can alter thelight generated by a light source and emitted towards a material of theobject so as to generate contrast. A common example is a fluorochromewhich stains perfusion, permeability or specifically binds to a certaintarget in the object, like target tissue, target cells or certain cellcomponents, like proteins, and which exhibits an interaction with light(UV, VIS and/or IR wavelength ranges) resulting in a specific absorptionand/or fluorescence. The concept of use of a marker substance is tohighlight one or more tissue characteristics which are altered at apresence of a disease. The marker substance is also called biomarker,probe or contrast agent. It is selected by the skilled person independence on the binding properties and the spectral propertiesthereof. In particular, the marker substance is selected such it targetsand reveals a molecular, structural, functional or compositional featureof the tissue which specifically changes in a gradual manner during thedisease progress. The presence of the marker substance preferably altersthe optical properties of the tissue, e.g. fluorescence or absorbance,in a way that the detected optical signal could even reveal the progressof the disease. The object preferably includes one or more markersubstances. If multiple different marker substances are provided, theypreferably have different spectroscopic properties. Besidesfluorochromes, the marker substances can be absorbing dyes,nanoparticles, polarization shifting moieties, fluorescence resonanceenergy transfer molecules, Raman particles etc.

The invention claimed is:
 1. A system for endoscopic imaging comprising:a light source configured to generate light; an insertion portionconfigured to be inserted into an object and comprising a distal end, aproximal end and a light guiding path configured to guide the generatedlight to the distal end of the insertion portion and to guide lightemanating from a medium within the object to the proximal end of theinsertion portion; a first imaging device mounted at the proximal end ofthe insertion portion and optically coupled to the light guiding path,the first imaging device comprising a plurality of first detectingelements, the first detecting elements exhibiting a first sensitivity tolight and being configured to detect a first part of the light emanatingfrom the medium to obtain at least one first image of the mediumexhibiting a first spatial resolution, the at least one first image ofthe medium comprising at least one of a color image or reflectance imageof the medium; a flexible guiding portion comprising a distal end and aproximal end, the distal end of the guiding portion being opticallycoupled to the light guiding path, and being configured to guide asecond part of the light emanating from the medium from the distal endof the guiding portion to the proximal end (5 a) of the guiding portion;a second imaging device provided at the proximal end of the flexibleguiding portion and comprising a plurality of second detecting elements,the second detecting elements exhibiting a second sensitivity to lightand being configured to detect the second part of the light emanatingfrom the medium to obtain at least one second image of the mediumexhibiting a second spatial resolution, the at least one second image ofthe medium being a fluorescence image of the medium, the secondsensitivity of the second detecting elements being higher than the firstsensitivity of the first detecting elements and the first spatialresolution of the first image being higher than the second spatialresolution of the second image; and a control unit configured to deriveat least one third image of the medium based on image data of the atleast one first image of the medium and image data of the at least onesecond image of the medium by correcting image data of the at least onesecond image of the medium for motion of the insertion portion and themedium relative to each other based on image data of two subsequentlyacquired first images of the medium, wherein the motion-corrected imagedata of the at least one second image form the at least one third image.2. The system according to claim 1, the control unit being furtherconfigured to determine, based on image data of the at least one firstimage of the medium, a two-dimensional motion field characterizing avelocity of the medium relative to the insertion portion or velocitiesof different regions of the medium relative to each other.
 3. The systemaccording to claim 2, the control unit being further configured torevert a motion-induced relative displacement of image data of the atleast one second image based on the two dimensional motion field toobtain the at least one third image of the medium.
 4. The systemaccording to claim 2, the control unit being further configured torevert a motion-induced relative displacement of image data in each oftwo or more successively obtained second images based on thetwo-dimensional motion field to obtain two or more adapted secondimages, and to combine the two or more adapted second images to obtainthe at least one third image of the medium.
 5. The system according toclaim 2, the two-dimensional motion field comprising motion-inducedvector components, including a distance and direction, of every pictureelement between the two subsequently acquired first images.
 6. Thesystem according to claim 2, the control unit being further configuredto weight image data of the second image with a factor the magnitude ofwhich depending on how much recent image data of the second image is,wherein image data of a more recently obtained second image is weightedwith a higher factor than image data of a less recently obtained secondimage.
 7. The system according to claim 1, the control unit beingfurther configured to determine, based on image data of at least twosuccessively obtained first images of the medium, a two-dimensionalmotion field, to revert a motion-induced relative displacement of imagedata in each of at least two successively obtained second images basedon the two-dimensional motion field to obtain two or more adapted secondimages, and to combine the two or more adapted successively obtainedsecond images to obtain the at least one third image of the medium. 8.The system according to claim 1, the control unit being configured tocontrol the first imaging device and the second imaging device to detectthe first part of the light emanating from the medium and the secondpart of the light emanating from the medium simultaneously.
 9. Thesystem according to claim 1, wherein the first part of the light and thesecond part of the light emanate from the same region of interest of themedium or from overlapping regions of interest of the medium.
 10. Thesystem of claim 1, wherein the insertion portion is rigid.
 11. A methodfor processing images generated by a system for endoscopic imaging,wherein the system comprises: a light source configured to generatelight; an insertion portion configured to be inserted into an object andcomprising a distal end, a proximal end and a light guiding pathconfigured to guide the generated light to the distal end of theinsertion portion and to guide light emanating from a medium within theobject to the proximal end of the insertion portion; a first imagingdevice mounted at the proximal end of the insertion portion andoptically coupled to the light guiding path, the first imaging devicecomprising a plurality of first detecting elements, the first detectingelements exhibiting a first sensitivity to light and being configured todetect a first part of the light emanating from the medium to obtain atleast one first image of the medium exhibiting a first spatialresolution, the at least one first image of the medium comprising atleast one of a color image or a reflectance image of the medium; aflexible guiding portion comprising a distal end and a proximal end, thedistal end of the guiding portion being optically coupled to the lightguiding path, and being configured to guide a second part of the lightemanating from the medium from the distal end of the guiding portion tothe proximal end of the guiding portion; and a second imaging deviceprovided at the proximal end of the flexible guiding portion andcomprising a plurality of second detecting elements, the seconddetecting elements exhibiting a second sensitivity to light and beingconfigured to detect the second part of the light emanating from themedium to obtain at least one second image of the medium exhibiting asecond spatial resolution, the at least one second image of the mediumbeing a fluorescence image of the medium, the second sensitivity of thesecond detecting elements being higher than the first sensitivity of thefirst detecting elements and the first spatial resolution of the firstimage being higher than the second spatial resolution of the secondimage; and wherein the method comprises: deriving at least one thirdimage of the medium based on image data of the at least one first imageof the medium and image data of the at least one second image of themedium by correcting image data of the at least one second image of themedium for motion of the insertion portion and the medium relative toeach other based on image data of two subsequently acquired first imagesof the medium, wherein motion-corrected image data of the at least onesecond image form the at least one third image.
 12. The method of claim11, wherein the insertion portion is rigid.